{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LLaMA Guidance\n",
    "\n",
    "This notebook shows how to control the <a href=\"https://ai.facebook.com/blog/large-language-model-llama-meta-ai/\">LLaMA</a> model using the guidance library. Note that this notebook uses a <a href=\"https://huggingface.co/huggyllama/llama-7b\">Transformers version of the model</a>, so please check out the special license terms noted on the HuggingFace model page before downloading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eecb6ddb77d34f2e8dd5afa8f0f89743",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-04-25 14:09:30.517932: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.1/lib64\n",
      "2023-04-25 14:09:31.019828: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.1/lib64\n",
      "2023-04-25 14:09:31.019892: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.1/lib64\n",
      "2023-04-25 14:09:31.019898: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.\n"
     ]
    }
   ],
   "source": [
    "import guidance\n",
    "\n",
    "# replace your_path with a version of the LLaMA model\n",
    "guidance.llm = guidance.llms.transformers.LLaMA(\"your_path/llama-7b\", device=\"cpu\") # use cuda for GPU if you have >27GB of VRAM"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A basic example\n",
    "\n",
    "Note that we have changed the anachronism detection example here to use more detailed guidance than we used for the GPT 3.5 example. This is because the 7B LLaMA model is much smaller and so we need more guidance to get it to answer how we want. Since we are using the Transformers version of the model, we can rely on <a href=\"../guidance_acceleration.ipynb\">guidance accelaration</a> to speed up inference as well using this guidance (as opposed to the OpenAI API which would slow down with detailed guidance since it does not yet support guidance acceleration)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-4df6086c-6795-40ee-94e9-f167fccc85d5\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-4df6086c-6795-40ee-94e9-f167fccc85d5\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>How to solve anachronism problems:\n",
       "Below we demonstrate how to test for an anachronism (i.e. whether it could have happened or not based on the time periods associated with the entities).\n",
       "----<span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{~#each examples}}\n",
       "Sentence: {{this.input}}\n",
       "Entities and dates:{{#each this.entities}}\n",
       "{{this.entity}}: {{this.time}}{{/each}}\n",
       "Reasoning: {{this.reasoning}}\n",
       "Anachronism: {{this.answer}}\n",
       "---\n",
       "{{~/each}}'>\n",
       "Sentence: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.input}}'>I wrote about shakespeare</span>\n",
       "Entities and dates:<span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#each this.entities}}\n",
       "{{this.entity}}: {{this.time}}{{/each}}'>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.entity}}'>I</span>: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.time}}'>present</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.entity}}'>Shakespeare</span>: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.time}}'>16th century</span></span>\n",
       "Reasoning: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.reasoning}}'>I can write about Shakespeare because he lived in the past with respect to me.</span>\n",
       "Anachronism: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.answer}}'>No</span>\n",
       "---\n",
       "Sentence: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.input}}'>Shakespeare wrote about me</span>\n",
       "Entities and dates:<span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#each this.entities}}\n",
       "{{this.entity}}: {{this.time}}{{/each}}'>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.entity}}'>Shakespeare</span>: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.time}}'>16th century</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.entity}}'>I</span>: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.time}}'>present</span></span>\n",
       "Reasoning: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.reasoning}}'>Shakespeare cannot have written about me, because he died before I was born</span>\n",
       "Anachronism: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.answer}}'>Yes</span>\n",
       "---\n",
       "Sentence: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.input}}'>A Roman emperor patted me on the back</span>\n",
       "Entities and dates:<span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#each this.entities}}\n",
       "{{this.entity}}: {{this.time}}{{/each}}'>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.entity}}'>Roman emperor</span>: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.time}}'>1st-5th century</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.entity}}'>I</span>: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.time}}'>present</span></span>\n",
       "Reasoning: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.reasoning}}'>A Roman emperor cannot have patted me on the back, because he died before I was born</span>\n",
       "Anachronism: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{this.answer}}'>Yes</span>\n",
       "---</span>\n",
       "Sentence: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{input}}'>The T-rex bit my dog</span>\n",
       "Entities and dates:<span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;entities&#x27; stop=&quot;\\nReasoning:&quot;}}\n",
       "{{gen &#x27;this.entity&#x27; stop=&quot;:&quot;}}: {{gen &#x27;this.time&#x27; stop=&quot;\\n&quot;}}{{/geneach}}'>\n",
       "<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.entity&#x27; stop=&quot;:&quot;}}'>T-rex</span>: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.time&#x27; stop=&quot;\\n&quot;}}'>65-68 million years ago</span>\n",
       "<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.entity&#x27; stop=&quot;:&quot;}}'>Dog</span>: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.time&#x27; stop=&quot;\\n&quot;}}'>10,000 years ago</span></span>\n",
       "Reasoning:<span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &quot;reasoning&quot; stop=&quot;\\n&quot;}}'> The T-rex cannot have bit my dog, because it died before my dog was born</span>\n",
       "Anachronism: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{#select &quot;answer&quot;}}Yes{{or}}No{{/select}}'>Yes</span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"4df6086c-6795-40ee-94e9-f167fccc85d5\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# define the few shot examples\n",
    "examples = [\n",
    "    {'input': 'I wrote about shakespeare',\n",
    "    'entities': [{'entity': 'I', 'time': 'present'}, {'entity': 'Shakespeare', 'time': '16th century'}],\n",
    "    'reasoning': 'I can write about Shakespeare because he lived in the past with respect to me.',\n",
    "    'answer': 'No'},\n",
    "    {'input': 'Shakespeare wrote about me',\n",
    "    'entities': [{'entity': 'Shakespeare', 'time': '16th century'}, {'entity': 'I', 'time': 'present'}],\n",
    "    'reasoning': 'Shakespeare cannot have written about me, because he died before I was born',\n",
    "    'answer': 'Yes'},\n",
    "    {'input': 'A Roman emperor patted me on the back',\n",
    "    'entities': [{'entity': 'Roman emperor', 'time': '1st-5th century'}, {'entity': 'I', 'time': 'present'}],\n",
    "    'reasoning': 'A Roman emperor cannot have patted me on the back, because he died before I was born',\n",
    "    'answer': 'Yes'}\n",
    "]\n",
    "\n",
    "# define the guidance program\n",
    "structure_prompt = guidance(\n",
    "'''How to solve anachronism problems:\n",
    "Below we demonstrate how to test for an anachronism (i.e. whether it could have happened or not based on the time periods associated with the entities).\n",
    "----\n",
    "\n",
    "{{~! display the few-shot examples ~}}\n",
    "{{~#each examples}}\n",
    "Sentence: {{this.input}}\n",
    "Entities and dates:{{#each this.entities}}\n",
    "{{this.entity}}: {{this.time}}{{/each}}\n",
    "Reasoning: {{this.reasoning}}\n",
    "Anachronism: {{this.answer}}\n",
    "---\n",
    "{{~/each}}\n",
    "\n",
    "{{~! place the real question at the end }}\n",
    "Sentence: {{input}}\n",
    "Entities and dates:{{#geneach 'entities' stop=\"\\\\nReasoning:\"}}\n",
    "{{gen 'this.entity' stop=\":\"}}: {{gen 'this.time' stop=\"\\\\n\"}}{{/geneach}}\n",
    "Reasoning:{{gen \"reasoning\" stop=\"\\\\n\"}}\n",
    "Anachronism: {{#select \"answer\"}}Yes{{or}}No{{/select}}''')\n",
    "\n",
    "out = structure_prompt(examples=examples, input='The T-rex bit my dog')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'entity': 'T-rex', 'time': '65-68 million years ago'},\n",
       " {'entity': 'Dog', 'time': '10,000 years ago'}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the entities generated are in the output\n",
    "out[\"entities\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' The T-rex cannot have bit my dog, because it died before my dog was born'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...as is the reasoning\n",
    "out[\"reasoning\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Yes'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ...and the answer\n",
    "out[\"answer\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height: 1px; opacity: 0.5; border: none; background: #cccccc;\">\n",
    "<div style=\"text-align: center; opacity: 0.5\">Have an idea for more helpful examples? Pull requests that add to this documentation notebook are encouraged!</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adatest",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
